Neural cortex

ABSTRACT

A neural network system includes a random access memory (RAM); and an index-based weightless neural network with a columnar topography; wherein patterns of binary connections and values of output nodes&#39; activities are stored in the RAM. Information is processed by pattern recognition using the neural network by storing a plurality of output patterns to be recognized in a pattern index; accepting an input pattern and dividing the input pattern into a plurality of components; and processing each component according to the pattern index to identify a recognized output pattern corresponding to the input pattern.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to a pattern recognition device that has anindex-based weightless neural network with a columnar topography, and toa method whereby an input pattern is divided into a plurality ofcomponents and each component is processed to identify a recognizedoutput pattern corresponding to the input pattern.

An artificial neural network is a structure composed of a number ofinterconnected units typically referred to as artificial neurons. Eachunit has an input/output characteristic and implements a localcomputation or function. The output of any unit can be determined by itsinput/output characteristic and its interconnection with other units.Typically the unit input/output characteristics are relatively simple.

There are three major problems associated with artificial neuralnetworks, namely: (a) scaling and hardware size practical limits; (b)network topology; and (c) training. The scaling and hardware sizeproblem arises because there is a relationship between applicationcomplexity and artificial neural network size, such that scaling toaccommodate a high resolution image may require hardware resources whichexceed practical limits.

The network topology problem arises due to the fact that, although theoverall function or functionality achieved is determined by the networktopology, there are no clear rules or design guidelines for arbitraryapplication.

The training problem arises because training is difficult to accomplish.

The n-Tuple Classifier has been proposed in an attempt to address theseproblems. This classifier was the first suggested RAM-based neuralnetwork concept. The first hardware implementation of the n-TupleConcept was the WISARD system developed at Brunel University around 1979(see “Computer Vision Systems for Industry: Comparisons,” appearing asChapter 10 in “Computer Vision Systems for Industry,” I. Alexander, T.Stonham and B. Wilkie, 1982). The WISARD system belongs to the class ofRAM-based weightless neural networks. This style of neural networkaddresses the problem of massive computations-based training by writingthe data into a RAM-network and the problem of topology by suggesting auniversal RAM-based network structure. However, the network topology ofWISARD-type universal structure leads to inefficient use of memory withthe consequence that the problem of scaling still remains acute withinRAM-based neural networks, and the application range of theWISARD-technology is limited.

Another example of neural networks that overcomes the problem oftraining by a simple memorization task is Sparse Distributed Memory (P.Kanerva, 1998, “Sparse Distributed Memory,” Cambridge, Mass.: MITPress). However, a problem with the Sparse Distributed Memory, as withthe WISARD system, is a large memory size. Another disadvantage of theSparse Distributed Memory is its computational complexity. This isbecause for this type of memory, an input word must be compared to allmemory locations.

N-Tuple classification systems use a method of recognition whereby aninput to the neural network is divided into a number of components(n-Tuples) with each component addressing a respective one of a seriesof component look-up tables. Each memory cell of the RAM stores adiscriminator word that has as many bits in it as there are patternclasses. Whenever a pattern is present at the input, each componentaddresses just one of its discriminator words. The network totals thenumber of 1s addressed in each bit position and recognizes the patternas the class that gives the biggest score (see “Computer Vision Systemsfor Industry: Comparisons,” appearing as Chapter 10 in “Computer VisionSystems for Industry,” I. Alexander, T. Stonham and B. Wilkie, 1982,page 189).

The presence of a number of look-up tables results in a potentiallylarge memory size. The memory size required is proportional to thenumber of components which the network may identify. This can result ina substantial increase in memory where the pattern size increases. Forexample, an artificial neural network might be designed for an imageprocessing application, initially using an n×n image, where n=128. Thisis a relatively low-resolution image by today's standards. Where theimage to be processed increases from n=128 to n=2048 , the size of thenetwork increases by a factor of 256. This increase in memory results inthe requirement for network expansion potentially requiring additionalhardware modular blocks. Where the resolution of the image increases apoint is quickly reached where the scaling to accommodate a highresolution image is beyond a practically achievable memory limit.

An object of the present invention is to address, overcome or alleviatesome or all of the disadvantages present in the prior art.

BRIEF SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided aRAM-based weightless neural network device for classifying an inputpattern as corresponding to one of a plurality of output patterns,comprising an input arranged to decompose an input pattern into aplurality of components, and RAM storing, for each component, aplurality of output pattern indications defining a column correspondingto the component, and having address lines so as to form a look-uptable, the address lines being addressed by each of the components,characterized in that the RAM comprises a RAM for storing a plurality ofaddress entities defining a linked data representation to represent thecomponent, and in that a logic device is provided for causing sequentialaccessing of the plurality of address entries.

The n-Tuple Classifier of the prior art requires the height of columns(i.e., the number of output pattern class indicators they store) to beflat and predetermined. For instance, if just a few columns are tall(i.e., containing many output pattern indications) whereas most columnsare short, then the allocated memory is mostly wasted. In contrast, thevariable columnar height of embodiments of the invention, achieved bythe use of the linked data structure, takes up less total memory.

Preferably, the neural network hardware component is a computer hardwarecomponent.

In a preferred form the neural network hardware component has potentialfor scaling. Scaling may be achieved in any suitable manner. It ispreferred that systematic expansion is achieved by increasing the sizeof the RAM.

The neural network may be trained in any suitable manner. It ispreferred that the neural network is trained by writing of data into theRAM and network topology emerges during the training.

It is preferred that performance of the neural network is adjustable bychanging decomposition style of input data, and thereby changing dynamicrange of input components.

It is preferred that input components to the neural network address asingle common index.

According to a second aspect of the invention, there is provided amethod of processing an input pattern by pattern recognition using aneural network storing an index to identify a recognized output patterncorresponding to the input pattern, the method including the steps of—

-   -   (a) storing a plurality of output pattern indications, each        corresponding to an output pattern to be recognized;    -   (b) accepting an input pattern;    -   (c) dividing the input pattern into a plurality of components;        and    -   (d) processing each component according to the index, the        processing comprising, for each component:        -   (1) looking up, in RAM, a first link associated with a first            output pattern;        -   (2) using the first link to look up, in RAM, a second link            associated with a second output pattern; and        -   (3) for each link looked up, increasing a corresponding            stored score corresponding to the associated output pattern.

Preferably each output pattern is divided into a plurality of recognizedcomponents with each recognized component being stored in the patternindex for recognition. The index preferably consists of columns witheach column corresponding to one or more recognized components.Preferably the index is divided into a number of columns which is equalto or less than the number of recognized components. More preferably,the index is divided into a number of columns which is equal to thenumber of recognized components.

The method may further include the steps of each input component beingcompared to the corresponding recognized component column, and a scorebeing allocated to one or more recognized components. Preferably thescore for each recognized component of a pattern is added and therecognized pattern with the highest score is identified as the outputpattern.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention will now be described in further detail by reference tothe attached drawings which show an example form of the invention. It isto be understood that the specificity of the following description doesnot limit the generality of the foregoing disclosure.

FIG. 1 is an index table illustrating recognition processing accordingto one embodiment of the invention.

FIG. 2 is a schematic block diagram illustrating a read cycle processaccording to that embodiment of the invention.

FIG. 3 is a schematic block diagram illustrating processing of a writecycle process output according to that embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention can be implemented through the use of a neural card builtwith the use of standard digital chips. The invention is an index-basedweightless neural network with a columnar topology that stores in RAMthe indications of output patterns. The network offers:

-   -   (a) Scaling potential: Systematic expansion of the neural        network can be achieved not by adding extra modular building        blocks as in previous artificial neural networks, but by        increasing the RAM size to include additional columns or by        increasing the height of the index. For example, 16 million        connections can be implemented with a 64 MB RAM.    -   (b) The required memory size is reduced by a factor of N, when        compared with previous n-Tuple systems such as the WISARD        system, with N being the number of input components (n-Tuples).    -   (c) The network topology emerges automatically during the        training.    -   (d) Training is reduced to writing of data into RAM.    -   (e) The performance can easily be adjusted by changing the        dynamic range of input components, which can be achieved by        changing the decomposition style of input data.

A device made according to the present embodiment is herein referred toas a Neural Cortex by the inventors. A general view of the single commonindex on which the present invention is based can best be described as acollection of vertical columns, wherein the signals propagate in abottom-to-top fashion.

Unlike traditional RAM-based neural networks, the present embodimentoperates by creating and memorizing an index (a linked datarepresentation) of input components. This index is created on training.

On retrieval, the present embodiment, like the prior art n-TupleClassifier, sums up the number of times each output class pattern wasactivated by input components. The summing operation provides thegeneralizing ability typical of neural networks. However, unlike theprior art n-Tuple Classifier, the present embodiment uses a singlecommon storage. In the case of the prior art n-Tuple Classifier, eachinput component (n-tuple) addresses its own look-up table. In the caseof the described embodiment, all input components address a singlecommon index. This brings about a dramatic decrease in memory size. Theabsence of a single common index in both the n-Tuple Classifier and theSparse Distributed Memory systems explains why previous RAM-based neuralnetworks had difficulties in terms of memory requirements whose largesize significantly limited the application range.

Further, a single common index according to a preferred and describedembodiment is an efficient solution to the neural network expansionproblem. As has been indicated above, both traditional artificial neuralnetworks and traditional RAM-based artificial neural networks havescaling difficulties when the application size grows. For instance, ifthe image size grows from 128×128 pixels to 2048×2048, then atraditional artificial neural networks will need a 256-fold increase inmemory because the number of n-tuples increases by the factor of256=2048*2048/128*128. However, paradoxically, in the same situation,the memory size according to the present embodiment may remain unchangedbecause still only one common index is used.

The present embodiment creates a single pattern index (104) of inputcomponents. The index contains the output components and is created bystoring the output pattern and training the neurons to recognize thepattern stored within the pattern index.

An output pattern S is decomposed into N number of components S₁, S₂,S_(N) (each having a reference number), such that each component S_(i)is interpreted as the address of a column (106 ₁, 106 ₂, . . . 106 ₁, .. . 106 _(N)) from the index (104). Each column stores entries (107 ₁,107 ₂) containing the reference number of those patterns which have thevalue S_(i) in one or more of their components; each column does notcontain more than one sample of each reference number. When an input I(100) is received, this is divided into a number of components I₁, I₂, .. . ,I_(X). Each input component I₁, to I_(X) is processed by thenetwork by inputting the component with the pattern index (104) via anaddress selector (102). Where a component of the input I_(i) produces acomponent of the output S_(i), then each reference number listed (107 ₁,107 ₂) in the column (106 _(i)) of S_(i) has a score of one added to itstotal. This process is repeated for each of the input components. Thescores are then added to determine the winner. The winner is thereference number with the greatest score. The reference number,corresponding to a recognized output pattern, is recognized by thenetwork.

An example of the device containing the pattern index is illustrated inFIG. 1. This figure illustrates where the device has been trained orprogrammed to recognize three words “background,” “variable” and“mouse.” In this figure the words are assigned the reference numbers 1,2 and 3 respectively. The output patterns are letters from “a” to “z”with these included as columns (106) within the index (104). When aninput (100) is received by the network, each of the components of theinput is processed by reference to this single pattern index. In thisexample the input is in the form of the word “mouse.” This input (100)is subsequently broken down into five letters (the input components).Each letter is processed in the network by using the index (104). Thenature of processing undertaken in the network can ensure thatprocessing of each component is undertaken virtually simultaneously. Thefollowing processing is undertaken—

-   -   (a) the component of the input “m” is processed and in this case        one point is added to the score attributable to reference number        3 (corresponding to output pattern “mouse”);    -   (b) the component input “o” is processed and one point is added        to the scores attributable to reference numbers 1 and 3        (corresponding to output patterns “background” and “mouse”);    -   (c) the component input “u” is processed and one point is added        to the scores attributable to reference numbers 1 and 3        (corresponding to output patterns “background” and “mouse”);    -   (d) the component input “s” is processed and one point is added        to the scores attributable to reference number 3 (corresponding        to output pattern “mouse”);    -   (e) the component input “e” is processed and one point is added        to the scores attributable to reference numbers 2 and 3        (corresponding to output patterns “variable” and “mouse”).

The network then sums up (not shown in this figure) the pointsattributable to each variable. In this instance variable 1 has a scoreof 2, variable 2 a score of 1 and variable 3 has a score of 5. Thevariable with the highest score is determined to be the winner and henceidentified. The variable 3 which has a score of 5, corresponding to theword “mouse,” is, therefore, considered to be identified.

In the case of standard RAM according to the prior art, two differentaddress words (i.e., input pattern components) always point towards twodifferent memory locations. This is no longer true in the case of thedescribed embodiment which uses a single index. For example, if theinput pattern has three components (a, b, c) and the component dynamicrange is 1 byte then the patterns (a, c, b), (b, a, c), (b, c, a), (c,a, b), (c, b, a) will produce the same score equal to 3 because thedevice according to the present embodiment is invariant with respect topermutations. The invariance is caused by the fact that all inputcomponents (n-tuples) address a single common index, thus creating thepermutational invariance, which is the price to be paid for dramaticreduction in memory size as compared to traditional RAM-based neuralnetworks. This invariance becomes practically harmless when thecomponent dynamic range is increased. For the above example, byincreasing the component dynamic range from 1 byte to 2 bytes, where thepattern (a, b, c) is converted into the 2 component pattern (ab, bc),the following scores will be obtained: (a, b, c)=>2, (a, c, b)=>0, (b,a, c)=>0, (b, c, a)=>1, (c, a, b)=>1, (c, b, a)=>0, so that the pattern(a, b, c) will be identified correctly.

In general case, the conversion of the n-component input pattern (s₁,s₂, . . . , s_(N)) into a new pattern (c₁, c₂, . . . , c_(M)) whosecomponents have a greater dynamic range and M<N is preferably done bythe software driver of the card of the present embodiment.

A read-cycle block-diagram of the embodiment is shown in FIG. 2. Theblocks “Roots” (110), “Links” (120), “Names” (130), “Score” (140) areRAM-devices. Σ(150) is a summer. T-logic (160) is a terminating logicaldevice.

-   -   1. Each pattern component (A-Word) is passed to the address bus        of the “Roots” RAM (110).    -   2. The output value R of the “Roots” RAM (110) is passed to the        address bus of the “Links” RAM (120).    -   3. The output value L of the “Links” RAM (120) is passed to the        address bus of the “Names” RAM (130).    -   4. And, finally, the output value N of the “Names” RAM (130) is        passed to the address bus of the “Score” RAM (140).

If L is 0 then the T-logic (160) terminates the process. Otherwise, the“Score” RAM (140) content found at address N that is determined by theoutput of the “Name” RAM is incremented by the value of 1. Next, the“Links” RAM (120) output is fed back to the “Links” RAM (120) addressbus. The process repeats itself from point 3.

A corresponding write-cycle block-diagram is shown in FIG. 3. CU is thecontrol unit (170).

-   -   1. Each pattern component A is passed to the address bus of the        “Roots” RAM (110).    -   2. The output value R of the “Roots” RAM (110) is passed to the        address-bus of the “Links” RAM (120).    -   3. The output value L of the “Links” RAM (120) is passed to the        address-bus of the “Names” RAM (130). The output value of the        “Names” RAM (130) is denoted by N, and the current pattern name        by P.    -   4. The values R, L, N and P are passed to the control unit 170,        which utilizes the following logic. If L is 0 then the control        unit makes a decision (point 5) on updating “Roots,” “Links,”        and “Names” RAM (130). Otherwise, L is fed back to the “Links”        RAM (120) address bus. The process repeats itself from point 3.    -   5. Decision Logic:        -   a) if N=P, terminate the process;            -   if R=0, increment the counter value C by 1,                -   write C to “Roots” RAM (110) at address A,                -   write C to “Links” RAM (120) at address R,                -   write P to “Names” RAM (130) at address L,            -   if R>0 & L=0, increment the counter (180) value C by 1,                -   write C to “Links” RAM (120) at address R,                -   write P to “Names” RAM (130) at address L,        -   b) terminate the process.

Performance of the embodiment can be adjusted in terms of memory sizeand read/write times. Normally, storage and recall times increase whenthe number of classes grows as the training continues. Additionalclasses increase the amount of reference numbers that are stored inindex columns and, therefore, the amount of index cells that have to beaccessed. As a remedy, one can increase the dynamic range D of inputpattern components. This increases the number of index columns becausethe index address space is equal to D. As a result, the same amount ofreference numbers will be spread upon the greater area, which, in turn,decreases the average index height H.

The processing time on storage and recall is proportional to the numberof accessed memory cells, which is proportional to HN. Here, N is thenumber of the pattern components. As D increases, the processing timeapproaches O(N). This follows from the fact that H is inverselyproportional to D.

The memory size is proportional to HD. However, H grows/decreases fasterthan D. Hence, adjusting the dynamic range D can efficiently control thememory size. In the worst case, the memory size does not exceed CD,where C is the number of pattern classes, which is because theembodiment has only one “look-up-table.” On the other hand, the memorysize of a traditional RAM-based artificial neural network is CDN becausefor this type of artificial neural network the number of inputlook-up-tables is equal to the number N of input pattern components.

It is to be understood that various modifications, alterations and/oradditions may be made to the parts previously described withoutdeparting from the ambit of the present invention.

1. A method of processing an input pattern by pattern recognition usinga neural network storing an index to identify at least one outputpattern as a recognized output pattern corresponding to the inputpattern, the method including the acts of: storing a plurality of outputpattern indications, each corresponding to an output pattern to berecognized; accepting an input pattern; dividing the input pattern intoa plurality of input components; processing each input component of theplurality according to the index, the processing comprising: looking up,in a RAM, a first link associated with a first output pattern; using thefirst link to look up, in the RAM, a second link associated with asecond output pattern; and for each of the first link and the secondlink looked up, increasing a corresponding stored score corresponding tothe associated first output pattern and the associated second outputpattern, respectively; and using each input component of the pluralityof input components as the address of a corresponding output patterncolumn wherein the stored score is allocated to one or more recognizedoutput patterns that are found in the corresponding output patterncolumns; outputting as the recognized output pattern the output patternwith a highest score.
 2. The method according to claim 1, wherein eachof the at least one output pattern is divided into a plurality ofrecognized components with each recognized component being stored in theindex for recognition.
 3. The method according to claim 2, wherein theindex consists of columns with each column corresponding to one or morerecognized components, each column being represented with a plurality oflinked index values.
 4. The method according to claim 3, wherein theindex is divided into a number of columns which is equal to or less thanthe number of recognized components.
 5. The method according to claim 3,wherein the index is divided into a number of columns which is equal tothe number of recognized components.
 6. The method according to any oneof claims 2 to 4, wherein each input component of the plurality of inputcomponents is used as the address of a corresponding output patterncolumn and a score is allocated to one or more recognized outputpatterns that are found in the corresponding output pattern columns. 7.The method according to claim 6, wherein the output pattern with thehighest score is identified as the recognized output pattern.